Allan B. Wilson; abwilson@awasu.com
Version 1.30; 28-Oct-05
The only change in this release is support for Awasu’s DownloadUrl parameter. By specifying a value for this parameter, Awasu will download the specified URL, rather than having the plugin do the download itself. If you are happy with your existing .ini files and the way WebScrape works, you can continue to use it just like you did before. However, if you have experienced problems with sites that rely on cookies—which the plugin does not handle—you should set DownloadUrl and any related values: DownloadUrl_UserName and DownloadUrl_Password if the site requires authentication, and, if necessary for your Website/URL, DownloadUrl_UtcOffset (see below).
This Channel Plug-in reads a URL page and—based on user supplied text patterns—extracts relevant information and outputs the results as RSS. Using this plug-in, Awasu can provide a “synthetic” news feed for sites that don’t provide their own feed(s).
There is no charge for personal or non-profit use. For commercial use or any required enhancements, please e-mail the author.
This is a standard Awasu channel plug-in. Simply create a new channel in Awasu and point to WebScrape.exe. You need to supply one parameter: the name of a Windows-style .INI (configuration) file. The referenced file must define symbols in the ChannelParameters section, as in the following example:
[ChannelParameters] URL=http://www.ucomics.com/calvinandhobbes UTCoffset= BaseURL= Title=Calvin and Hobbes Description=Cartoon retrieved from Website MaxItems=1 Shorthand= SectionPattern= ItemPattern-1=<font class="comictitle">(?P<T>.*?)</font> ItemPattern-2=.*?(?P<D><img src="(?P<L />.*?)") width ItemPattern-3=
The various symbols are defined below. Note that some are defined in terms of regular expressions. These are text patterns that are used to match text on the scraped page. Unfortunately, regular expressions (REs) can be tricky to get right: first, because the syntax can be quite intricate; second, because you must understand the page to be scraped well enough to specify the exact patterns necessary to identify and extract news items.
While there are literally entire books on writing regular expressions, here are some key items you should find useful when scraping pages:
<font class="comictitle">
matches exactly those characters on the Calvin and Hobbes Webpage. You should View Source in your browser to see the contents of the Web page you want to scrape.
(?P<T>), (?P<L>), and (?P<D>)
corresponding to RSS item title, link, and description, respectively. They can be in any order on the Webpage, but you must assign each of them a value. The plug-in combines your ItemPatterns into a single (long) pattern and applies it repeatedly to match multiple items on the Webpage. You can use several ItemPatterns simply to “break up” the process of writing what otherwise would be one long string; you may include as many as you want/need, so long as they are numbered sequentially and you specify the required title, link, and description assignments. If any of the three is blank (for example, if there is no link), you must still define it, but assign it a null value: e.g., “(?P<L>)” (without the quotation marks).
<font class="comictitle">
all the following characters up to (but not including) the closing “font” tag are assigned as the item title.
URL is the URL of the page to scrape for information.
Because some Web pages are tied to date/time of creation, WebScrape provides for dynamically generated URLs based on date, time, and offset from UTC (the former GMT—Greenwich Mean Time).
Here are some common symbols that can be embedded in the URL string to refer to selected date/time items:
%d Day of the month as a decimal number [01,31] %w Weekday as a decimal number [0,6]; Sunday is 0 %j Day of the year as a decimal number [001,366] %I Hour (12-hour clock) as a decimal number [01,12] %H Hour (24-hour clock) as a decimal number [00,23] %m Month as a decimal number [01,12] %M Minute as a decimal number [00,59] %S Second as a decimal number [00,61]; 61 accommodates leap- and double leap-seconds %W Week number of the year (Monday as the first day of the week) as a decimal number [00,53]; all days in a new year preceding the first Monday are considered to be in week 0 %U Week number of the year (Sunday as the first day of the week) as a decimal number [00,53]; all days in a new year preceding the first Sunday are considered to be in week 0 %y 2-digit year (without century) as a decimal number [00,99] %Y 4-digit year (with century) as a decimal number [0001,9999]
(The cryptic nature of the symbols is due to their implementation in a standard C-language library for manipulating dates and times.) As a quick example, if news you want in RSS is at “http://www.newssite.com/directory/subdir/YYYY-MM-DD/news.html”; (where YYYY represents the 4-digit year, MM the numeric month, and DD the numeric day) you would specify URL as “http://www.newssite.com/directory/subdir/%Y-%m-%d/news.html”; (without the quotation marks). All times are UTC (again, the former GMT); see UTCoffset and DownloadUrl_UtcOffset, below.
UTCoffset (or DownloadUrl_UtcOffset if using DownloadUrl) must be defined if you use date/time parameters in URL or DownloadUrl, as explained above. The UTC offset is the number of minutes of the site specified by URL/DownloadUrl from UTC (+ for east of UTC, – for west). For example, the west coast of the US is -8:00 from UTC. Therefore if the site you are accessing is in the US Pacific timezone, the UTC offset could be set to -480, which is -(8 * 60); note that this doesn’t take into account Daylight Savings Time, so you should consider using the offset between UTC and current local time (-7:00 or -420 for Pacific Daylight Time).
Note: The symbols in the table above are used in URL as described when specified in the .ini file. However, if you are using DownloadUrl, you need to use the symbols in “Awasu style”, that is, instead of just the single % character before the letter, you need one after it as well. Therefore, rather than, say, %Y in URL you would use %Y% in DownloadUrl.
BaseURL can be defined if the page specified by URL contains relative (as opposed to absolute) URLs. If you supply a value for this symbol, the plug-in will convert relative URLs to absolute URLs; otherwise, embedded URLs on the page won’t work.
Title Whatever name you want to give to the channel
Description However you wish to describe the channel
MaxItems The plug-in will not create more than this many news items in the output RSS feed, even if there would be more to match on the URL page.
Shorthand This is an optional feature. It allows you to define “abbreviations” for selected characters. Again, it is optional—you can simply leave it empty if you wish. However, should you wish to define, say, “~” to mean “.*?” you could specify Shorthand as follows:
Shorthand=~=.*?
Other abbreviations can be be added on the same line, preceded by one or more spaces. Use the “=” sign to separate the shorthand character from its definition.
SectionPattern This is another optional feature. If the URL page you are scraping has several different types of items—for example, hardware news and software news, or national news and international news—and all items have the same text pattern regardless of which “section” they are in, you can use SectionPattern to isolate, say, the hardware news so that only hardware items are reported in the RSS feed; otherwise you will see both hardware and software items displayed. By defining SectionPattern so that it matches a subset of the URL page, the plug-in will create RSS items from within that section of the page, that is, the ItemPatterns will be applied only to the resulting section.
ItemPattern-n As explained before, the text strings defined on these lines are combined into a single string, which is used to match individual news items on the selected page (or, if SectionPattern was defined, within the defined section of the page). You must define three named matches:
(?P<T>), (?P<L>), and (?P<D>)
corresponding to RSS title, link, and description, respectively. (Should you wish to set any of them empty, you can use “(?P<x>)”—without the quotation marks—so, for example, “(?P<D>)” would set the item description to nothing.)
The description above covers the original WebScrape implementation—and is still valid. However, this public release of WebScrape expands its capabilities. Specifically, WebScrape now supports multiple patterns (and sections) within the retrieved URL page. Therefore if a page has several different ways of representing news items you would like to scrape, you can add additional information in the .ini file. Please refer to the (artificial) example below:
[ChannelParameters] URL=http://www.example.com BaseURL=http://www.example.com/ Title=WebScraped News Description=RSS news extracted from Web page MaxItems=25 [MultiSearch] Section1Pattern=Section 1 Pattern text ItemPattern1-1=Item Pattern 1 text 1 ItemPattern1-2=Item Pattern 1 text 2 Section2Pattern=Section 2 Pattern text ItemPattern2-1=Item Pattern 2 text 1 ItemPattern2-2=Item Pattern 2 text 2 Section3Pattern= ItemPattern3-1=Item Pattern 3 text 1 ItemPattern3-2=Item Pattern 3 text 2
Note that there is now a new section (“[MultiSearch]”) in the .ini file. This lets WebScrape know you want the expanded functionality. Simply include as many SectionMPattern and ItemPatternM-n elements as you wish; M (and n for each M) must start at 1, and increase by one (i.e., with no gaps). As before, if you want specific patterns applied against the entire Web page, just leave the SectionMPattern empty (such as Section3Pattern above); otherwise you use the section patterns to isolate text to match against their respective patterns.
There are two ways to write and debug your scripts:
1) You can create an .INI file (using, say, Notepad) with the various symbols and their values, then simply run WebScrape.exe in a command window, supplying the name of your .INI file as the first argument. RSS output will be produced in the command window. By examining the generated RSS you can determine if your patterns were correct or not.
2) Use WebScrapeSettings.exe, an included GUI-based utility, to make the process of defining and testing the regular expressions easier.
Once you are happy with the generated RSS, you can create a new channel in Awasu based on the plug-in and specify your .INI file in the setup dialog. From then on, you can’t tell the difference between a “real” feed and your scraped one!
You should know that Web page structure changes as designers and Webmasters make updates or adopt new conventions. Therefore, your carefully crafted scraping could stop working someday and you will need to update your patterns to compensate.
Packaged with the plug-in are a number of .INI files that you can try out. By studying them while using your browser to view the HTML source of the corresponding URL page, you should be able to see how to write your own .INI files. Once you have gained some experience and confidence, you will be able to make Awasu even more valuable by extending its reach to sites that don’t have RSS feeds.
Thanks to users of earlier versions of this plug-in, we have begun collecting and making new .INI files available for download from awasu.com; everyone benefits from the community’s efforts.
Please note the Warning above: Web sites do change, so some of the supplied .INI files may no longer work.
The plug-in was written in Python and the resulting code “frozen” to create the standalone executable file.